{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64e3755a",
   "metadata": {},
   "source": [
    "## Dynamic Graph Encoder\n",
    "The input is first put through a gated graph convolution unit from PyTorch Geometric. <br />\n",
    "Its output is fed into as standard pytorch LSTM unit. <br />\n",
    "\n",
    "The output of the DyGrEncoder is the last hidden state of the last LSTM layer. <br />\n",
    "\n",
    "In this example, the features of a single time step are expanded with the conv unit <br />\n",
    "and then sequentially integrated using the LSTM.<br />\n",
    "\n",
    "The H, C inputs to the DyGrEncoder are used to initialize the LSTM states. <br />\n",
    "The hidden and cell state are expanded internally to (1, ...), meaning one LSTM layer. <br />\n",
    "Therefore, more than one LSTM layer will fail to execute unless the initial states are _None_. <br />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fea8010",
   "metadata": {},
   "source": [
    "### GyGrEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7e473b3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200/200 [06:29<00:00,  1.95s/it]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/50lEQVR4nO3dd3xV9f348dc7e++EAAlJ2HsGVERFCwra4lZsHbjo0lbtUL9trT87rLXD2lot7r1rRcWJC2RI2FtCICSQkEVC9vz8/jgn4SZkXCB3hPt+Ph73cc+657xzkpz3PZ/P+Xw+YoxBKaWU7/LzdABKKaU8SxOBUkr5OE0ESinl4zQRKKWUj9NEoJRSPk4TgVJK+ThNBEr1MSLyuYgYEVng6VjUyUETgfIaIrLXvsBd5OlYvIGIzLTPx94Oq94A/gFsc39U6mQU4OkAlFIgIoHGmEZntjXG/MvV8SjfoncEqs8QkYtFZI2IVIpIrog8IiIx9rogEXlcRApFpF5E8kTkHXudiMgf7WX19jYfikh8F8cJF5EHRWS3iFSJyAYRucZeN1hEWkSkVEQC7WVp9jf3UjuOABH5pYhsF5FqEdkmIgsd9n+vvf0bIvKaiNQC3+sQw0zgM3u2df/GXteuaEhEnrHnnxOR90WkVkQ+suN6045hpYhkOOx/rIi8JyJFIlJsbzeoF35Nqg/SRKD6BBE5H/gvMN5+rwR+BLxib3ItcBNQAjwJrAWm2+u+BdwNNNvrvgTGAZFdHO5p4Of29q8Bw4DnROQqY0wO8BUQB8yyt7/Cfn/VGNMA/A54ABDgJSAE+I+IXNfhOJcCQ4DngcIO6/KBN+3pSqyioH90EW+rq4EqoAyYDWwEYoAc4FQ7LkQk2T4Hs4HlwOfAJcCHIhLcwzHUSUgTgeorbrHf/2iMuQ6YCTQB54nIcCDQXr8ZeBG4Hkiyl7Wuy8a6sN8CDAT2dTyIiCQBl9uzs40xNwD/Z8/far8/Z79fab+3JoLnREQcYl0BVANb7PkfdjhcDnCKMWahMeYDxxXGmGygtQiozBhzmzHmto7xdvCpMeZy4HF7vhbrYt8a/yT7/RogFut87AP2A8XASODsHo6hTkJaR6D6inT7fTuAMaZEREqAZCAN6+I8E7gQmA8Y4BMRuRj4CPg31gWwtbglC5gHFHRxnFpjTK49vcN+T7PfXwMeBi4SkZFAJvCNMWaViCQCEfZ213fY99AO818bY5p6+sGPwXb7vdx+zzbGtIhIpT0fbr+n2++j7Fd3MSofoHcEqq/Ya7+PBLDL9xPsZblAkzHmSiAK6+L2Cda34UsAf6xv6TFYF7rnsC7eN3VznFCHMvMRDsfBGFMBvA1EA4vsda13CSVYdwEAE4wxYowRrP+1zA7Hqu/hZ2623539P23uYb7VXvv9rdb47Bj7YxWdKR+jiUB5owdEZJXD60zgEXvd/4nIM1jl2gHAx8aYb4CrRGQ7Vvn+T7HqAMD6djwd2INVZHQHcLrDunaMMUVYj2cCfCwiTwF/tOcdn9ZpvfCfgXX38bz9eeMQ60d2BfbLWMVA9x7TWYA8+z1FRJ4QkTuP8fNdeRHrZ7/YrjT/j4h8Yh+vXy8dQ/UhWjSkvNHwDvNxxpj/icgVwF3AZVgVov/BqgQG2In1bfx8rErgAuD3wLtYdwG7sCqNY+ztHuPIt/mObsC6KF6MVQ+wG/ibMeYlh20+xKrgTQa+MMY41jf8GigFFmBV4B4G1gOvOvnzA2CM2Ssif8G6c7kR2IpVCX1CjDEHROQs4A/ANGAGVl3BI1jnRvkY0YFplFLKt2nRkFJK+ThNBEop5eM0ESillI/TRKCUUj6uzz01lJCQYNLT0z0dhlJK9Slr164tMcYkdrauzyWC9PR0srKyPB2GUkr1KSKS29U6LRpSSikfp4lAKaV8nCYCpZTycZoIlFLKx2kiUEopH6eJQCmlfJxLE4GIzBGRnSKSLSJ3dbJ+kIh8JiLrRWSTPRyhUkopN3JZIhARf6xubecCo7H6ix/dYbNfA68ZYyZhjSr1b1fFk7W3jAc+2IH2tqqUUu258o5gGtZQeTn2gN6vYA0j6MhgjSgF1mhPB1wVzOb9FTz6+W5KqxtcdQillOqTXJkIBnJkhCWAfHuZo3uBq0UkH1jCkcHB2xGRhSKSJSJZxcXFxxXM4ERrGNmc4uoetlRKKd/i6criq4BnjDEpWCNLPS8iR8VkjFlkjMk0xmQmJnbaVUaPBidY43bvKak6gXCVUurk48pEsB9IdZhPsZc5uhF4DcAYsxII4ciA5L1qQEwoQQF+5JToHYFSSjlyZSJYAwwTkQwRCcKqDF7cYZt9WOPIIiKjsBLB8ZX99MDfT0iPD9OiIaWU6sBlicAY0wTcgjXI93asp4O2ish9IjLP3uxnwM0ishF4GVhgXPhYT0ZCOHv0jkAppdpxaTfUxpglWJXAjsvucZjeBpzuyhgcZSRE8OmOIppbDP5+4q7DKqWUV/N0ZbFbDU4Mp7HZkH+oxtOhKKWU1/CtRGA/OaQVxkopdYRPJYKM1kSgFcZKKdXGpxJBXHgQ0aGB7C7WtgRKKdXKpxKBiDC8XwS7DlZ6OhSllPIaPpUIAEYkR7KjsFI7n1NKKZvvJYJ+kVTWNVF4uM7ToSillFfwuUQwvF8kADsK2xcPrdlbxvaCw54ISSmlPMrnEsGIZCsRfOOQCIwx/PjFdfzp/R2eCksppTzGpS2LvVFMWBD9ooLZ6ZAI9pRUU1RZT1RorQcjU0opz/C5OwKAEclR7HR4cmhVThkAB8pr21UiF2k9glLKB/hmIugXwa6iKppbrIv+6j2lANQ0NFNR2wjAh1sLmfbHpXyy7aDH4lRKKXfwyUQwdmA0DU0tbMg7hDGGVTmlRAZbpWT7y2upqm/it29vBeCJ5TmeDFUppVzOJxPBOSOTCArw452NBewtreHg4XrmjksG4EB5Hf9cuouDlXVcML4/q3LK2FGoTxMppU5ePpkIIkMCOWdEEu9uKuCp5XsAuCLTGkztQHktS7YUMGtUP35/4ViCA/x4dkWuJ8NVSimX8slEADBv4gBKqup5flUuC6anM3lQLEEBfmw9UEFeWS1T0mKJDQ/i3DHJfLajyNPhKqWUy/hsIjhnZBIRwQEMigvjl3NG4OcnDIgO4WO7cnh8SjQAE1KiKTxcR3FlvSfDVUopl/G5dgStQgL9efr6qcSHBxEWZJ2G/tGh7C2tQQTGDbQSQev7lv0VnD0yyWPxKqWUq7j0jkBE5ojIThHJFpG7Oln/dxHZYL++EZFyV8bT0dT0OAYnRrTND4gJBawBbCJDAgEYMzAaEdi8v8KdoSmllNu47I5ARPyBR4DZQD6wRkQW2+MUA2CMud1h+1uBSa6KxxkDY0IAmJAa07YsIjiAjIRwTQRKqZOWK+8IpgHZxpgcY0wD8ApwYTfbXwW87MJ4etR6RzAhJabd8nEDo9miiUApdZJyZSIYCOQ5zOfby44iImlABvBpF+sXikiWiGQVFxf3eqCtRvWPwt9POGVwXLvl4wZGU1BRR0mVVhgrpU4+3vLU0HzgDWNMc2crjTGLjDGZxpjMxMRElwUxITWG9ffMZmRyVLvlYwZYFcYb88pddmyllPIUVyaC/UCqw3yKvawz8/FwsVCrKLuS2NGkQTFEBgfw/pZCD0SklFKu5cpEsAYYJiIZIhKEdbFf3HEjERkJxAIrXRjLCQkJ9GfO2GQ+2FJIWXUDv317CznFVZ4OSymleoXLEoExpgm4BfgQ2A68ZozZKiL3icg8h03nA68YLx9E+OJJA6mqb+Lyx1bw7MpcXly9z9MhKaVUr3BpgzJjzBJgSYdl93SYv9eVMfSWUwbHkxwVwu7iakIC/fhsZxG/+fZoT4ellFInzFsqi72ev59wx7nDuTIzlV+cN5Kc4mpyS6s9HZZSSp0wTQTH4IrMVB64bDyzRlldTWhndEqpk4EmguOQFh/O4MRwPt3pujYNSinlLpoIjtOsUf1YkV1CYYWOa6yU6ts0ERynq09Jo8UYnlmx19OhKKXUCdFEcJwGxYcxd2x/XlydS1V9k6fDUUqp46aJ4ATcfOZgKuuaeHtDVw2mlVLK+2kiOAETU2NIjAxmXW65p0NRSqnjpongBI0dEKVdVCul+jRNBCdo3MBodhVVUtvQacepSinl9bpNBCLiLyLzRGSEuwLqa8YMjKbFwPbCw54ORSmljku3icAeH+BJ4DT3hNP3tA5uv1WLh5RSfZQznc69CCwQkTVAQetCY0yZy6LqQ/pHhxAXHsTm/RU0Nbfg7yeIiKfDUkoppzlTR/AT4AxgE1Bsv7STHZuIMHZgNB9uPci4ez/i35/vxhjDAx/sYPmuEk+Hp5RSPXLmjuBLwKvHCvC0UzLiWLarmKTIYB77Yjdp8WE8+vluCsprmTEswdPhKaVUt3pMBMaYmW6Io09beOZgrshMpaCilnn/+orbX90AwL6yGs8GppRSTuixaEhEokXkGRE5aL+eEpFodwTXVwT6+5EYGcz4lBhmDE2gsdmQFBnMvrJaT4emlFI9cqZo6GHgGo4MPL8AEOB6F8XUp93zndG8vWE/IQH+/PXjb6htaCY0yN/TYSmlVJecqSyeC/zZGJNqjEkFHgQucGbnIjJHRHaKSLaI3NXFNleIyDYR2SoiLzkfunca3i+SX5w3krSEcADyDmnxkFLKux3PmMVOVRyLiD/wCDAbyAfWiMhiY8w2h22GAXcDpxtjDolI0nHE45UGxYUBsK+0htKqBhIjgxiaFOnhqJRS6mjOJIIlwC9E5Lv2/EDgOSc+Nw3INsbkAIjIK8CFwDaHbW4GHjHGHAIwxpw0j6WmxoYCkFNSxT+XZhMc6M87t55O/+hQD0emlFLtOVM0dBtWo7JQ+/U8cLsTnxsI5DnM59vLHA0HhovIVyKySkTmdLYjEVkoIlkiklVc3DeGh4wLDyI8yJ+31h+gsr6Jkqp6fvDCOhqbWzwdmlJKtdNjX0PAb4CnjTGJ9muBMaa8l44fAAwDZgJXAY+LSEzHjYwxi4wxmcaYzMTExF46tGuJCKlxYWwvOEygv/C7C8ewMa+clbtLPR2aUkq140xfQxcBQ45j3/uBVIf5FI48edQqH1hsjGk0xuwBvsFKDCeF1nqCqelxXDolhSB/P5Zna2tjpZR3caaO4HPgHhEJpn1fQ//t4XNrgGEikoGVAOYD3+2wzf+w7gSeFpEErKKiHKci7wNS7UQwc0QiYUEBTE6LYZl2O6GU8jLOJILW9gIP2++C9eRQtw/HG2OaROQW4EN726eMMVtF5D4gyxiz2F53rohsA5qBXxhjTpqykyGJEQCcM9J6GOqMYYk8+OFOiivrSYwM9mRoSinVxplE8P+Od+fGmCVYTx05LrvHYdoAd9ivk84lkwcyIjmi7bHRM4Yl8OCHO/kqu4SLJnWsN1dKKc/oNhHYlcVRwLvGmM/cE9LJIyTQnylpcW3zYwZEExMWyHJNBEopL+LKymLVgb+fMCk1Rsc4Vkp5FVdWFqtOjEiOYnl2CQ1NLQQF6JDRSinPc1llsercqP6RNDYbckqqGJkc5elwlFLKqURwHzowTa8ZkWxVHO8srNREoJTyCs4MTHOvG+LwGYMTIgj0F7YXVHLhRE9Ho5RS3VQWi8g6EZktIuH2YDQj7eUXi4gOXH+cggL8GJIYwc7Cw54ORSmlgO6fGpoIxAIhWIPRDLCXBwE6QtkJGJkcyY7CSk+HoZRSQM+9j2rdgAuM7B9FQUUdFTWNng5FKaV6TAQ3APdjJYRbRORhrLsDdQJaK4x32MVD2UWVWI2slVLK/XqqLD7PYfoih2m9ap2AUfbTQjsPVhIU4MfF/17BCzeewoxhCR6OTCnli7pLBGe7LQof0y8qmOjQQLYXVNLQZA1UsyHvkCYCpZRHdJkIjDFfuDMQXyIijEyOZGfhYarrmwDYrpXHSikP0T4OPMRKBJVsyi8HYHuBPk6qlPIMZ1oWKxcY2T+K6oZmqktriAgOYG9JNbUNzYQGac8dSin30jsCDxlpPzkE8J0JA2gx8M1BLR5SSrlfl3cEInJPV+sAjDH39X44vmN4vyOJ4Mqpqbz89T62FxxmQmqM54JSSvmk7oqG7nWYNli9jrZOg9UZnTpO4cEBpMWH0dxiGD8wmvAgf21trJTyiO4SweX2+9nAWcDfsYqSfgqsdGbnIjIH+AdWl9VPGGP+1GH9AuBBrMHtAf5ljHnC2eD7uhtOzwDAz08YkRzJNq0wVkp5QHePj74JICJ/Af5gjHnKnhfglz3t2B7m8hFgNpAPrBGRxcaYbR02fdUYc8txxt+nXTc9vW16fEoMr2Xl0dxi8PeTrj+klFK9zJnK4mDgtyJyr4jcB9wDhDnxuWlAtjEmxxjTALwCXHj8oZ7cJqbGUNPQrBXGSim3cyYR/ByIx0oAv7anf+bE5wYCeQ7z+fayji4VkU0i8oaIpHa2IxFZKCJZIpJVXFzsxKH7nkmDYgBYv6/co3EopXxPj4nAGPMSkI7V19BFQLox5pVeOv479v7GAx8Dz3YRwyJjTKYxJjMxMbGXDu1dBsWFERcexPp9hzwdilLKxzjbjmAqcA6QDZwrIhOc+Mx+wPEbfgpHKoUBMMaUGmPq7dkngClOxnPSEREmpcawPq/c06EopXxMj4lARG7D+uZ+K5AMXIL1pE9P1gDDRCRDRIKA+cDiDvvu7zA7D9juXNgnp0mDYsguqqKiVscpUEq5jzN3BLcBrzvMfwJM7ulDxpgm4BbgQ6wL/GvGmK0icp+IzLM3+4mIbBWRjcBP8PGxDiamxgK09T+klFLu4ExfQ7HARuAyez4Mq11Aj4wxS4AlHZbd4zB9N3C3U5H6gFH9rdbGuw5Wccawk7MuRCnlfZxJBF8DP7Snfw7MAL5yWUQ+LC48iMiQAPaWVns6FKWUD3GmaOhWoBari4k5QAFWcZHqZSJCenw4e0o0ESil3KfbOwK7dfBwrAriFnvxTmNMs6sD81XpCeFsyNNHSJVS7tPtHYF9wX8SyDTGbLNfmgRcKCM+jP2HatuGsFRKKVdzpmjoRWCBiIwRkbjWl6sD81XpCeG0GMg7VOPpUJRSPsKZyuKfYHU9vclhmXHys+oYpcWHA7C3pJohiREejkYp5QucuZh/yZExCJSLZSRYiUArjJVS7tJjIjDGzHRDHMoWGxZIlD5CqpRyox4TgT3+wHxgHBBiLzbGGGd6IFXHSETISAhnd1E1xhis06+UUq7jTNHQI8APOHq4Sk0ELjI0KZI31+Vz2v2fcsXUVG44PZ2YsCBPh6WUOkk589TQxcBL9vRPgc+A37ksIsVvvj2K+y8Zx+gBUTy8dBcX/3sFLS1aTaOUcg1nEkEssMyeLgDeABa6LCJFTFgQV00bxFMLpvLnS8ezp6Sa9drITCnlIs4kgkKsIqRCrDED/urk51QvmDMumUB/4cOtBz0dilLqJOXMBf3XwG6sOoE6oALta8htokICmT4kgQ+3FmKMFg8ppXqfM0NVvmCM+cAY84oxJtkY078Xh6pUTjhvTDK5pTXs1IHtlVIu4MwIZZ928lrqjuCUZdboJAA+31ns4UiUUicjZx4fndnJMi2jcKOkyBD6R4ews1DvCJRSvc+ZROA4VFYscC/W00PKjYb3i9REoJRyCWcqi43D6zCwE7jOmZ2LyBwR2Ski2SJyVzfbXSoiRkQyndmvLxqRHEl2cRVNzdo9tVKqdzlzR1DC0UVBO3v6kD2ozSPAbCAfWCMii40x2zpsF4nVUG21UxH7qOH9ImloaiG3rEZ7JVVK9apj7X20GdgL/MWJz00Dso0xOQAi8gpwIbCtw3a/Ax4AfuHEPn3WiH6tA9tXaiJQSvUqV/Y+OhDIc5jPB05x3EBEJgOpxpj3REQTQTeGJkUgAjsLq5gz1tPRKKVOJs70PvpUN6uNMebG4zmwiPgBfwMWOLHtQuxuLQYNGnQ8h+vzQoP8GRQXxjcd2hK0tBhE0F5KlVLHzZmioQUc3fOo43RXiWA/kOown2IvaxUJjAU+ty9iycBiEZlnjMly3JExZhGwCCAzM9NnH10d3i+yXaMyYwyz//4FF4wfwB2zh3swMqVUX+bMU0N/wep0bhZwrj39V2AqVj1AV9YAw0QkQ0SCsMY0WNy60hhTYYxJMMakG2PSgVXAUUlAHTEqOZI9JdUcrmsEYOfBSnYXV/Pm2nztfkIpddycSQTXAq8aYz41xnwCvAZcYYxZa4xZ29WHjDFNwC3Ah8B24DVjzFYRuU9E5vVG8L5mxrBEmlsMy3eVALS97y+vZVN+hSdDU0r1Yc4UDdUC94vIqVhFQvOAUmd2boxZAizpsOyeLrad6cw+fdnkQTFEhwby6Y4izh/Xn6+ySxgQHUJRZT1LthQwITXG0yEqpfogZ+4IbsJKBtcAVwM19jLlZgH+fpw5PJHPdxZR19jM6j1lzBrdj9OGxPPBFu2dVCl1fJzpfXQpkAZMtF/pxpjPXBuW6so5IxMpqWrg4aW7qGloZsbQBOaO7U9uaQ3bC7QLCqXUses2EdgD12OMaQD6Y7USPssNcakunDU8CX8/4d+f7yY00J9Th8Rz7ph++Al8sEW7gFJKHbsu6wjsrqYNMEtEbsR+fNNe91tjzO/dEJ/qIC48iOdumEZdYzPjUqKJCgkEYFpGHO9vKeSOc0d4OEKlVF/T3R3BWOA9e/oH9vvvgC+Am10ZlOre6UMT+NaofiRFhrQtmzu2P7uKqsgu0uIhpdSx6S4RRAOlIhINTAL2GWPuBZ4FktwQmzoGc8YmA/DepkIPR6KU6mu6SwR7scYpfsHe7gN7+SCcfHxUuU+/qBDOHJ7I0yv2cKi6wdPhKKX6kO4SwW+AEcAFWF1R/9VePh+rFbDyMr86fxSVdU389eMeewlXSqk2XSYCY8zrWD2IngIMNsZki0gA8F3gx26KTx2DEcmRXHNqGi+t3sfu4ipPh6OU6iO6fXzUGFNqjFljjKmy55uMMRuNMQfdE546VrecM5QAPz+eX5nr6VCUUn2EMy2LVR+SEBHM+eOSeWNtPlX1TZ4ORynVB2giOAldOz2dqvom3lqX7+lQlFJ9gCaCk9Ck1BgmpESzaFkODU3WYPd1jc08sSyH2oZmD0enlPI2PSYCERkhIo+LyMci8qn9WuqO4NTxERFunz2cvLJaXlmzD4BXvt7H79/bznubtRsKpVR7znRD/T+sx0gdaTeXXu6s4YmckhHHw0uz+c74ATz11V4A1uaWcdmUFM8Gp5TyKs4UDcUBf8fqdC7RfmnLYi8nItx9/igqahs496Ev2VdWQ1RIAGv2HvJ0aEopL+NMIngOGApEYN0JtL6Ul5uYGsPj12ZSWddIalwoN50xmOyiKm15rJRqx5mioZ9hXfi/7bDMOPlZ5WEzRyTx/k/PxE+goKIOgLW5h5g1up+HI1NKeQtnLuZfoncAfVpGQjgASZEhBPoLWZoIlFIOekwEJzKWsIjMAf4B+ANPGGP+1GH9D7C6q2gGqoCFxphtx3s81b3QIH/GDIhmxW5r0Pv/rd/P+n2H+PW3RxPor08SK+WrekwE9ihl84FxQGsH+MYY87MePucPPII1qlk+sEZEFne40L9kjHnM3n4e8DdgzjH/FMpp8yYM4L53t/H5ziLueXsLh+uaKK9tJDkqBH8/4ZdzRno6RKWUmzlTNPQI1sA0BhB7mcGqO+jONCDbGJMDICKvABcCbYnAGHPYYftwtAjK5eZPS+XhT3fxoxfXUdPQzCWTBvLf9fsB8PcT7pg9nAC9O1DKpzjzH38x8JI9/VPgM6yRynoyEMhzmM+3l7UjIj8Wkd3An4GfdLYjEVkoIlkiklVcXOzEoVVXwoICWDA9nZqGZmaNSuKvV0zgmeuncsfs4TS3GAoP13k6RKWUmzmTCGKBZfZ0AfAGsLC3AjDGPGKMGQLcCfy6i20WGWMyjTGZiYmJvXVon7VgejqzRvXjl3NGIiLMHJHElLRYAPLKaj0cnVLK3ZwpGiq0tysEngCCgMPdfsKyH0h1mE+xl3XlFeBRJ/arTlBMWBBPXJfZbllKbCgA+YdqgHgPRKWU8hRn7gh+DezGqhOoAyqA25z43BpgmIhkiEgQVoXzYscNRGSYw+wFwC4n9qtcoH90KCKQf0jvCJTyNc48PvoCgIjEAGnGmHpndmyMaRKRW4APsR4ffcoYs1VE7gOyjDGLgVtEZBbQCBwCrju+H0OdqKAAP/pHhZB3qMbToSil3MyZx0fTgdeBScAcEfkN8IUx5p6ePmuMWQIs6bDsHofpnx5rwMp1UmLDnLojePnrfQxJjGBaRpwbolJKuZozRUOPYT3tI0ALVkvj+a4MSnlGSmwo+3tIBP9bv5+7/7uZfyz9xk1RKaVczZlEMB34l8P8bqyKX3WSSYkLo6Cilsbmlk7XZxdVctd/NyECG/MqaG7RZh9KnQycSQQlwFh7OgnrbuCAyyJSHpMSG0qLgYJyqy3BNwcr2V5wuC0xvLh6Hy0G7pozkqr6JnYVVXoyXKVUL3EmETyOdfEX4EWsLiP+48qglGc4PkJa09DEd/65nLn/WMbMBz+noqaRj7cdZMbQBM4bkwzAutxyD0arlOotPSYCY8z9wPVYDcneBK43xjzo6sCU+6XGhgGQd6iGdbnl1De1cO1paewvr+XXb28h/1Ats0f3Iy0+jLjwINbt00FulDoZODWmgDHmWeBZF8eiPKx/dAhRIQF8vecQA2Jq8RP45ZyR7C2t4Z2NBxCBb41KQkSYlBrDek0ESp0UurwjEJHmbl5N7gxSuUeAvx+zRyfz8bZClmeXMHZgNBHBAfzwrCEATEqNISnS6oB2closu4urKa/R0c6U6uu6Kxpq7Wn0AFaPoY6v7S6OS3nIBeOTOVzXxPp95ZxitxM4dXAcC6an88OZQ9u2mzQoBoD1eeUeiFIp1Zu6SwRPA9VAArAZuMMYM6715ZbolNvNGJpIZIhVYjgtw+pzSES4d94YZjuMajYhJQY/gfX7yj0RplKqF3WZCIwxNwL9gR9hdR73gYjstUcdUyepoAA/Zo/uhwhMTY/tcrvw4ABGJEdpPYFSJ4FunxoyxlQDOcAeoAHr7iDSDXEpD7pzzkieum4qMWFB3W43eVAMG/aV06INy5Tq07qrLP6ViOwCPgWGArcC/Y0xr7srOOUZ/aJCOHtkUo/bTR4US2V9E7uKqtwQlVLKVbp7fPR3WENH5mC1Lp4HzLOGMMYYYy50fXjKm7VWGC/bVUxTSwtjBkR7NiCl1HHpqR2BAEPslyMtC1BkJIQTGxbI79+zHiJbfMvpjE+JcfrzTc0tVNc3Ex0W6KIIlVLO6K6OIKOb12DXh6a8nYhw55yRzJ9qDUS3o+DY+h5atCyHs/7yGXWNza4ITynlpC7vCIwxue4MRPVN86cN4vLMVN5cl8/ukmOrK1i5u5TymkZW7ynjrOE6FrVSnuJMp3NKdcvfT0iLD2dPcbXTn2lpMWywG6N9+U2xiyJTSjlDE4HqFYMTwtlTUs2ekmqm/O7jtot8V/aUVlNZ14S/n/SJRHDv4q3c9GyWp8NQyiVcmghEZI6I7BSRbBG5q5P1d4jINhHZJCJLRSTNlfEo18lIDCe3tIaPthZSWt3Asyv2drv9RjtRXDJpILuKqjhQ3vMQmZ60Pq+cNXvLPB2GUi7hskQgIv7AI8BcYDRwlYiM7rDZeiDTGDMeq5vrP7sqHuVaQxIiaGhu4fW1+QAs2VxARW1jl9tvzCsnPMifG2ZkAPCFl98VFJTXUlHbSEVN1z+TUn2VK+8IpgHZxpgcY0wD8ArQru2BMeYzY0yNPbsKHQKzz8pIDAcgu6iKiakx1De1sHjD/i6335BfwbiUaEYmRzIgOoRPdxQBsPVABaVV9W6J2VmNzS0U2zHlljlfD6Jc61+f7uLhpbs8HcZJwZWJYCCQ5zCfby/ryo3A+52tEJGFIpIlIlnFxd79zdFXZSSEt03fdEYGYwZE8VpWfqfbHiivZfuBw0xIiUFEmDW6H8t2FVNQUculj67g/vd3uCtspxw8XIexW87sK6vpfmPlNm+t38+HWws9HcZJwSsqi0XkaiAT6HTkM2PMImNMpjEmMzFRHzP0RvHhQUTZvZaeOjieiyYOZPP+CvaWtP8GXdvQzMLnswgK8ONKu/3B7NH9qGts4aevbKCusYXlu0owxnvaLBZW1LVN55ZqIvAGzS2GvLJayrWorle4MhHsx+q1tFWKvawdEZkF/AqYZ4zxrjIB5TQRYWhSBCOTI0mICGbuOGtc4/c2F7Tb7p63t7D1wGH+MX8igxMjADglI57IkAC+3lNGkL8fhYfryCnxniKYAw6JYJ8mAq9QeLiOhuYWyqp1YKTe4MpEsAYYJiIZIhIEzAcWO24gIpOA/2AlgSIXxqLc4IFLx/PPqyYBkBIbxqRBMSxxSATvbjrA62vz+fHMoXxr1JGxDYIC/Dh7hNXJ3e2zhwOwIrvEjZF3r7DCeqJpRL9IrSPwErml1u+htrFZW6b3ApclAmNME3AL8CHWiGavGWO2ish9IjLP3uxBIAJ4XUQ2iMjiLnan+oBh/SIZ1u9IL+UXjOvP1gOHeS0rjyeW5XDXm5uZmBrDT2cNO+qzN8zI4PIpKdx8RgYDY0L5KrvUnaF360B5HRHBAYwZEKV3BF7CsYhOi4dOnFOD1x8vY8wSYEmHZfc4TM9y5fGVZ82bMIAnlu3hl29sAuD0ofE8cOl4Av2P/v4xMTWGiakxAEwfEs9H2w7S3GLw95OjtnW3woo6+keHkBoXxlsb9lPf1ExwgL+nw/JpjongUE0DydEhHoym73NpIlC+LSkqhOV3ns2WA4fxE5zumfSM4Ym8vjafNXvLOHWwNVzm8l0lBAX4Mc0eR7k31Dc1szm/gsz07vdZUFFLcnQIafFhGAN5ZbUMTYrotTjUsWstGgI4pPUEJ8wrnhpSJ68Afz8mpsYcU/fUs0YlERbkz9t2O4TahmZ++MJavvv4Kj7Y0vPjgsYY/vzBDr7/fPddQvzt42+47LGV5JXV0NTcwsrdnRdHFVTUMSA6lLT4MKD9RUh5Rm5pDQNjQgE4pEVDJ0wTgfI6YUEBzBmTzLubCqhrbObDrYVU1jfRPyaEW15ax/aCw+w6WMm0P3xyVJ9GLS2GPy7Zzr8/382HWw9y8HDdUesPlNdSVt3A8yutDnY3769g8cYDXPX4Krbsr2i3fUOT1ZgsOTqEEclR+PsJ6/e1P6ZyL2MM+8pqmJBqDYR0qEbvCE6UJgLllS6aNJDKuiY+21HEa1l5pMaFsvjHMwgK8OPJ5XtY9GUORZX1/OvT7LbP7C6u4uJHV/D4sj2cPtQqUlqXe6jdfv/y0U6m/+lT5i9aSW1jM34C2w4cJsvebv2+9tsXVVqNyQbEhBARHMCElGhW7PaeJ5p8UWl1A1X1TW11SuWaCE6YJgLllaYPiad/dAi3v7aBFbtLuXxKKrHhQVw2JYXFGw7w9sYDRIcGsnTHQXYXV2GM4Y5XN5BbWs3fr5zAUwumEhTgxzqHC3tBRS1PLt/D4MRwdhdX8+3xAxiWFMm2gsNt3/I35LW/I3jdbh3d2uZh+pAENuZXUFmnxRGe0lpRPDQpgvAgf8qq9XdxojQRKK8U4O/HyzefyrwJA0iPD+PyTKsbquump9PQ3EJDUwuLrplCoL8fj36+m1U5ZWzMr+AX543g4kkpBAf4M25gNOscinH+8ckujIFnr5/G8jvP5sHLxjNmQBQb8sr55qA1utrG/CPbf76ziIc/3cUlkweSmRYLWAmqucW4rCfS9fsO8Znd79L9S7ZzwcPL+GhroVe1tPa01jqatPhwYsKC9I6gF+hTQ8prpSeE8+fLJrRbNiQxgosnDcQYwymD47nutDQeX7aHz3cWkRARxKWTj/RbOHlQDM+uzOVwXSN/en8Hr6zJ48YZGaTGhbVtM3pAFP9db1VKT0iNYVN+OZV1jYQHBXDv4q0MS4rgDxeNQ8R6jHVyWixBAX58lV3KOSP70Zuq6pu4+bm1lFTVc9W0Qbz89T6iQgJY+PxaHrt6MnPG9u/V4/VVuaU1iEBKbCix4YFaR9AL9I5A9Tl/v3IiD823WjDfNXcUV2SmUFLVwILp6YQEHnm+f/KgWBqaWjjv71/y0up9/HDmEO6aO7Ldvkb3j2qbvu60NIyxKo+XZ5ewt7SGH589lNCgI/sMCfQnMy2Wj7cdpKq+qVd/rkVf5lBSVc/I5Ehe/nofE1KiWf1/swgO8GNth7oOX5ZbWs2A6FCCA/yJDQvy6qeGjDHUN3l/y2dNBKpP8/cT/nTJeF648RS+f9aQdusmp8UiAvVNLTxz/VTunDPyqMZsowdYiSA1LrStm4uNeRU8vyqX+PAg5oxNPuqYP5w5hP3ltfzg+bXUNvTOP3lJVT2Pf5nDBeP68+r3T+MHZw3hke9NJjTIn4wEq07DU2oamryqaCq3rKbtUV4rEXjvHcHbGw4w7Q9Le/1LQ2/TRKD6PD8/YcawhKMu8v2iQnjpplP54KdnMNO+yHcUExbEkMRwTs2IJzY8iPT4MP756S6Wbj/IlVNTO21BfMawRP50yTiWZ5cw7Y+f8Ku3NpNTXHVCP8OSzQXUNjbzk28NIzo0kLvmjiQl1rrYDUmKYPcJ7v94lVbVM+OBz7j91Q1ekwxySx0TQaBXNyjbUVhJRW0jm/Mret7YgzQRqJPaaUPiSYrqvvuB175/GvfOGwPAX6+YyIUTB5KZHse1p6V3+ZnLM1N57funMXtUP15fm8+3/vYFz6/ce9xxvr+5kKFJEYxIjjxq3dDECPLKajzSudpjX+ymrLqB/204wJPL97j12KVV9Ty3ci8tLUcSUGVdI2XVDaTFW+NfxIYHcbiuiabmFrfG5qwSe0Ajx4cQvJFWFiufFx8R3DY9JS2WKfYTQj2ZlhHHtIw47j5/FD9/fSP3vbuNCcfYihqgrLqB1XtK+dHMoZ2uH5IUQYuxvgl3lihOxKqcUtLiw+gfHXrUusKKOp5dmcslkwdS29DM/e/v4NTB8YwdGN2rMXTl+VW5PPTJLoYlRXLaEKtdSOujo2lxR4qGACpqG9v9Hr1FayLY5OWJQO8IlDpBiZHB/GP+RBIjgrn15fXH/M39422FtBg6rY8AGGIPA9pd8dBnO4vaHoF11trcMuYvWsVp93/KT15ef1TRz0urc2luMdw+azh/umQ8ceFB3Pnmpnbfvgsr6vj1/zZT09D7ZeAr7C4/3tt8oG1ZayIYZBcNxYQFAt7burjtjiBPi4aUOunFhAXxl8snkFtaw1NfHVsRyrubCkiNC2XMgKhO1w9OsBqz7S7qPBFU1DRy87NZXPrvFcf0dNE7GwuskeIyU1m88cBRn123r5yRyZGkxoURHRbIffPGsPXAYX7/3va2ZPDmunxeWLWPdzcVdHaI41bb0Mz6fYcQgQ+2FLYdr3U8iLaiIfuO4ESeHGpoaiHLRe1CSiobEIH95bUUV3rvuFuaCJTqJdOHJjB7dD/+/dnutm+CPdl1sJJlu0q4fEpqW1uFjkKD/BkYE9rlHcHH2w/S1GIIDvTnuqe+dqqBVXOLYcnmAs4ekchv540mOjSQx5fltK03xrApv7xdMdecscksmJ7OMyv2cvWTq2loauErewCht9YdNfhgl8d96JNvjhrCtKOs3DIamw1XZqZSUtXA13usC3VuSQ0JEUFEBFul2oMTw/H3E95a79zxO/PCqlwue2xljzEdK2MMpdX1TB5kFTV6c/GQJgKletHdc0dS19jMo5/vxhjDXz7cyfubC2hqbuHONzZx3zvbOOzQPcUTy/YQEujH1aemdbvfIUkRZHdIBBvzyjlc18gHWwrpHx3Co1dPpqq+qcteVMHqdO+DLYUs2VxAUWU9F4wfQFhQANecmsZH2w7y9ob9VNQ2sre0hsN1TUxMPVIfICLcO28Mv7toLKtyynh7w36ycg8RERzAqj2lHCiv7fH8bMwv56FPdnHDs2vaddPR0mL43hOrmPev5dy7eCtLNhcQ4Cf84rwRhAX588jn2ZTXNLAxv7ztbgCskfCun57OS6v3HXdbi+V2Mlu3r3fbahyubaKx2TBzeCJ+gld3VqiJQKleNDgxggvG9+e1NXks3V7Evz7L5taX13PTc1m8mpXH0yv2cO7fviS3tJrCijre2rCfSyenEBce1O1+R/ePYmdhJfmHaiisqOOmZ9dw4SNfccVjK/lyVzHnjUlmYmoMYUH+bWXrnfl4+0F+8MJabn15PcEBfnxrpPVY7bXT00iMCOanr2xg7kNfsjrH2kdnFd/fmzaIlNhQ7n9/Bw1NLfzs3OEYYz0z35PWJJVbWsPPXtvYVi+x5UAFX2WXUt/Ywourc3n56zwmpsYQHxHMb749mhW7Szn1/qXsPFjJZVNS2u3zttnD6R8dwv1Ltvd4/I4am1vaftaOPdm2qm1oPq72IsX2XeGg+DBOyYjnzXX5NHrp002aCJTqZQump1NZ38RPXllPUmQww/pF8vnOYm6akcFbPzqduqZmvv/8Wm54Zg3+Itx0xuAe93ntaWmICA9+uJMbnlnDV9mlXH96OntKqmloamHu2GQC/f2Ymh7Hypz2iaC5xbA5vwJjDO9sPEBceBA/nDmEX10winC7iCUpMoRld57Nw1dN4kBFHX/56BtCAv0Y1skAPH5+wlXTBlFW3UCAn3B5ZirTMuJ4cvmednc7nVm5u5SRyZHcOWeEfQdiJY9PdxQhAi/dfArv3noGZ49IbLtLumraIB66ciJpceE8tWAqV00b1G6fEcEBfHfaILJyDzlVJLcxr5wKu05hU3451Q3NBPn7sbGTRNDcYrhy0UpueGZNj/vtqDWWhIhgbj4zg4KKOt7rUJdy8HAdD33yDdc+9bVH6xBcmghEZI6I7BSRbBG5q5P1Z4rIOhFpEpHLXBmLUu4yaVAsE1JjqGlo5vtnDeG5G6bx58vGc/f5o5iYGsPD8yex82Al3xys5NGrJ5OREN7jPgfEhHLNqWm8veEA2woO88j3JvHb74zh6QVTuXFGRtsoa6cNiSe7qIoiexyGmoYmvv98Ft/513KeWLaHpduLmDs2mTvnjDyqnURwgD/zJgxgSlosJVX1jB0QTUAnw4oCXD4lBX8/YWJqDBHBAfzmgtGUVtfzt4++6fJnqG9qZs3eMk4bEs+NMwYzMTWG//fOVkqq6vlsZ3HbHcCI5Eievn4aF00a2PbZCycO5MPbz2xr/d1Ra4PBL78p7vY8VtY1ctljK/jH0l0ArLDHxr540kC2FRymrrGZ4sr6tjuVF1fnsim/gpU5pex3oujLUWsiiI8IYubwJIYkhrPoy5y2fdc1NnPpoyt46JNdfPlNMR9s7XnQJVdxWSIQEX/gEWAuMBq4SkRGd9hsH7AAeMlVcSjlCXfMHs6MoQl8d9ogEiODuSIztW385TOHJ/Lo9ybzzPXTumzx3JkfzRzCwJhQfnHeiLYO76YPTeA33x7dtu/p9vP2K3NKySur4Yr/rOTTHUWkx4fxhyXbqW1s5jsTBnR7nB/aXXV01x4iKSqE3180ljtmDwdgXEo015yaxnMr97ZVIHe0YV859U0tTB+SgL+f8OfLxlNd38yNz2axKb+cc47hXHQ0ZkAUCRFBfL6z+0SwOseqhM7KtSqfv9pdwuj+UZw9MpHGZsMv3tjE1D98whl//ozvP5/Fgx/sbOuP6r1NPRd9OSqpPHJH4OcnfP+sIWwrOMyiL61K+SeX7yH/UC3P3ziNgTGhLOshibmSKxuUTQOyjTE5ACLyCnAhsK11A2PMXnuddxacKXWczhqeyFnDE7tcfzw9icZHBLP8zrO7fLoIYMyAaCJDAvi//25GRBCBx6/NZFhSJOc+9AVRIYFM7WGM5nNGJvHzc4czd1z3MXYsovn5eSNYnVPGwueyeHnhqe0SSU1DE69m5eEntI07PbxfJH+7cgK3vrweY+DskcefCPz8hLOGJ7F0x0GaWwx1jc28uDqXWaP6tY0lAUcqhrcdOExRZR1rcw9x/ekZTLAHuXln4wGmD4knLCiAPSXV9I8J4Z/fncTtr27gnY0FLDxzSGeH71RJVQN+cuQR18unpPDFzmIe+GAHBRV1vJ6Vx+zR/ThjWCJnDk/k3Y0HaGxuOaqrFHdwZSIYCOQ5zOcDpxzPjkRkIbAQYNCgQT1srdTJq7skAFYnfA9fNYml2w9SVdfEbbOGk24XPT169RT8RNruHrri5yfccs6wY44tKiSQ526cxiX/XsGtL6/nkzvOor6phedW7uXJZXsorW7g8ikpRIcGtn3m2+MH0NDUwordpV22o3DWzBGJvLkun7v/u4msvYfIKanm4aXZ/PY7o5kzNpnIkEBW7C4hPMif6oZm/vVpNo3Nhtmj+9E/OpSBMaEEB/rxn2umEBkS2G7f3xk/gD8s2c7OwkqnW3eXVNUTFx7cdr5FhAcvH09xVT0vrd5HUlQwvzp/FABnDkvg5a/3sSGvvMdE7Qriqo6k7DL/OcaYm+z5a4BTjDG3dLLtM8C7xpg3etpvZmamycrqflBypZTnfLajiOufWcPdc0e21WmcNTyRW88Z2laX4QqVdY3c/FwWG/MqiA0L5O7zR/HE8j1szCsnwE+45rQ0nv5qL98/czD/+TKHAD8hOjSQr381C38/Iae4iqjQQBI66aqi6HAd5z70JdGhgbzxg+kkRrbf5nBdI6VVDe3qe256Nov8QzV8cNuZPcZeUdPIpN99xC1nD+WOc0ec+MnohIisNcZkdrbOlXcE+4FUh/kUe5lS6iQ2c0QiU9Njuf/9Hfj7CU9cm8ms0b07iE9nIkMCeWXhaW2d1Pn5CXPHJvP13jJeWr2Pp7/aC1h3IR9vO0hOSTXfGpXU9o3dsQipo6SoEJ5aMJXvPb6aa55czXM3TGvXmeFv397K+1sK+Pj2s9oGPiqpqj8qYXQlOiyQyYNieXlNHpdnprYbPMkdXFkYtQYYJiIZIhIEzAcWu/B4SikvICLcNXcU4UH+/OGisW5JAo78/AQ/++Ie4O/H9CEJ/POqSfzkW8OYMTSB0QOimGS39j13dOf9O3Vm8qBYHr82k31lNVxsF389uXwPpVX1vLepgLrGFn73blsVKCVV9Z3eXXTlDxePo6GphaufXO10y/Te4rKiIQAROR94CPAHnjLG/EFE7gOyjDGLRWQq8BYQC9QBhcaYMd3tU4uGlOobPFXx6YzPdhTx0NJdvLrw1Haj2jljU345d/93M4eqGzhQUccpGXGs3lPGpZNTeHNdPimxoRRV1tPQ1MLCMwfzf3Y9gDPW7TvE/EWrOH1IPE8tmNpjndCx6K5oyKWJwBU0ESilvEFTcwvzF60iK/cQ0zLieOHGU/jFGxtpbjF2xbM/l09JOeZinmdX7OW3i7dy+6zhXH3qIP70/g4276/glIw4LpuSyriU4+sGXBOBUkq5QP6hGn7wwlrumjOKGcMSemWfxhh+8MJaPtx6EBHwEyEzLZaN+eX88eJxXDI5peeddEITgVJK9SEtLYZl2SV8vrOIeRMGMGlQLPVNzRjDMRdltfLUU0NKKaWOg9VArn2jxM7Gz+6147lsz0oppfoETQRKKeXjNBEopZSP00SglFI+ThOBUkr5OE0ESinl4zQRKKWUj9NEoJRSPq7PtSwWkWIg9zg+mgB0PoaeZ2lcx8Zb4wLvjU3jOjbeGhecWGxpxphOh83rc4ngeIlIVlfNqz1J4zo23hoXeG9sGtex8da4wHWxadGQUkr5OE0ESinl43wpESzydABd0LiOjbfGBd4bm8Z1bLw1LnBRbD5TR6CUUqpzvnRHoJRSqhOaCJRSysed9IlAROaIyE4RyRaRuzwYR6qIfCYi20Rkq4j81F5+r4jsF5EN9ut8D8W3V0Q22zFk2cviRORjEdllv8e6OaYRDudlg4gcFpHbPHHOROQpESkSkS0Oyzo9P2J52P6b2yQikz0Q24MissM+/lsiEmMvTxeRWodz95ib4+rydycid9vnbKeInOfmuF51iGmviGywl7vzfHV1jXD935kx5qR9Af7AbmAwEARsBEZ7KJb+wGR7OhL4BhgN3Av83AvO1V4gocOyPwN32dN3AQ94+HdZCKR54pwBZwKTgS09nR/gfOB9QIBTgdUeiO1cIMCefsAhtnTH7TwQV6e/O/t/YSMQDGTY/7f+7oqrw/q/Avd44Hx1dY1w+d/ZyX5HMA3INsbkGGMagFeACz0RiDGmwBizzp6uBLYDAz0RyzG4EHjWnn4WuMhzofAtYLcx5nhalZ8wY8yXQFmHxV2dnwuB54xlFRAjIv3dGZsx5iNjTJM9uwo4vhHPezmublwIvGKMqTfG7AGysf5/3RqXiAhwBfCyK47dnW6uES7/OzvZE8FAIM9hPh8vuPiKSDowCVhtL7rFvrV7yt3FLw4M8JGIrBWRhfayfsaYAnu6EOjnmdAAmE/7f05vOGddnR9v+7u7AeubY6sMEVkvIl+IyBkeiKez3523nLMzgIPGmF0Oy9x+vjpcI1z+d3ayJwKvIyIRwJvAbcaYw8CjwBBgIlCAdVvqCTOMMZOBucCPReRMx5XGuhf1yLPGIhIEzANetxd5yzlr48nz0x0R+RXQBLxoLyoABhljJgF3AC+JSJQbQ/K6310HV9H+C4fbz1cn14g2rvo7O9kTwX4g1WE+xV7mESISiPULftEY818AY8xBY0yzMaYFeBwX3Q73xBiz334vAt6y4zjYeqtpvxd5Ijas5LTOGHPQjtErzhldnx+v+LsTkQXAt4Hv2RcQ7KKXUnt6LVZZ/HB3xdTN787j50xEAoBLgFdbl7n7fHV2jcANf2cneyJYAwwTkQz7W+V8YLEnArHLHp8Ethtj/uaw3LFM72JgS8fPuiG2cBGJbJ3GqmjcgnWurrM3uw54292x2dp9S/OGc2br6vwsBq61n+o4FahwuLV3CxGZA/wSmGeMqXFYnigi/vb0YGAYkOPGuLr63S0G5otIsIhk2HF97a64bLOAHcaY/NYF7jxfXV0jcMffmTtqwz35wqpZ/wYrk//Kg3HMwLql2wRssF/nA88Dm+3li4H+HohtMNYTGxuBra3nCYgHlgK7gE+AOA/EFg6UAtEOy9x+zrASUQHQiFUWe2NX5wfrKY5H7L+5zUCmB2LLxio/bv1be8ze9lL7d7wBWAd8x81xdfm7A35ln7OdwFx3xmUvfwb4QYdt3Xm+urpGuPzvTLuYUEopH3eyFw0ppZTqgSYCpZTycZoIlFLKx2kiUEopH6eJQCmlfJwmAuXz7B4mTYdXuQuOc6+978t6e99KnYgATweglBdZj9XTI0CDJwNRyp30jkCpI4qxGux8AiwVkQX2N/gX7L7oS0Tk560bi8jNdh/x1SLytYjMsJcHicj9IpJr92X/ZYfjnC3WWAHFInK5/ZnT7Y7Y6uzlbu/9UvkuTQRKHXEuVjIopn13GmdjdZZWCDwoIhNE5BysgcSLsTojGwQsFpF4rD7j78JqkXoLVotUR9+y9xcN/Mle9kusFt4/Bu4DSnr7h1OqK1o0pNQRq4Ff29OHgHH29FPGmP+ISBPwBHAW1oUf4LfGmI9FZBDwf1gDhHwHq6uAK43Vr3xHfzPGLBKRH2L1XQNW9wHfxupSYB1W1wFKuYXeESh1RIkx5hP7tdZhuXR4d2Q6vDujdVCUJo78D96J1fPlLqw+ebLEHl5SKVfTOwKljhggIvMd5gPt9+tFZB/wE3v+C6yOwH4G/D8RGYJ18T6ENRrYO0Am8KqIvAGMN8bc1sOx7wbqsYqT8rCGa4wCyk/wZ1KqR5oIlDpiEu0HJbndfl8K/AhIBn5hjNkIYI/k9kvgb8A24HZjTKmI/AkIBb4HnINz3Sm3ALfaxyjFGjN33wn/REo5QXsfVaoL9sAuT2Nd/P/i4XCUchmtI1BKKR+ndwRKKeXj9I5AKaV8nCYCpZTycZoIlFLKx2kiUEopH6eJQCmlfNz/B1rHXlZzewx8AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train MSE: 0.0775\n",
      "Test MSE: 0.9859\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch_geometric_temporal.nn.recurrent import DyGrEncoder\n",
    "\n",
    "from torch_geometric_temporal.dataset import ChickenpoxDatasetLoader\n",
    "from torch_geometric_temporal.signal import temporal_signal_split\n",
    "\n",
    "loader = ChickenpoxDatasetLoader()\n",
    "\n",
    "lags = 10\n",
    "epochs = 200\n",
    "\n",
    "dataset = loader.get_dataset(lags)\n",
    "\n",
    "train_dataset, test_dataset = temporal_signal_split(dataset, train_ratio=0.2)\n",
    "\n",
    "### MODEL DEFINITION\n",
    "class RecurrentGCN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(RecurrentGCN, self).__init__()\n",
    "\n",
    "        self.rnn = DyGrEncoder(\n",
    "            conv_out_channels=32, \n",
    "            conv_num_layers=1, \n",
    "            conv_aggr='mean', \n",
    "            lstm_out_channels=64, \n",
    "            lstm_num_layers=1\n",
    "        )\n",
    "        \n",
    "        self.out = nn.Sequential(\n",
    "            nn.Linear(64, 32),\n",
    "            nn.Dropout(0.2),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32, 1),\n",
    "            nn.Flatten(start_dim=0, end_dim=-1)\n",
    "        )\n",
    "\n",
    "    def forward(self, window, h, c):        \n",
    "        edge_index = window.edge_index\n",
    "        x = window.x\n",
    "\n",
    "        H, C = [], []\n",
    "        for t in range(lags):\n",
    "            x_t = x[:,t].unsqueeze(0).T\n",
    "            h_tilde, h, c = self.rnn(x_t, edge_index, None, h, c)\n",
    "            \n",
    "            H.append(h.detach())\n",
    "            C.append(c.detach())\n",
    "\n",
    "        pred = self.out(h_tilde)\n",
    "\n",
    "        return pred, H, C\n",
    "    \n",
    "model = RecurrentGCN()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n",
    "\n",
    "\n",
    "### TRAIN\n",
    "model.train()\n",
    "\n",
    "loss_history = []\n",
    "for _ in tqdm(range(epochs)):\n",
    "    total_loss = 0\n",
    "    h, c = None, None\n",
    "    for i, window in enumerate(train_dataset):\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        y_pred, H, C = model(window, h, c)\n",
    "\n",
    "        h = H[0]\n",
    "        c = C[0]\n",
    "    \n",
    "        assert y_pred.shape == window.y.shape\n",
    "        loss = torch.mean((y_pred - window.y)**2)\n",
    "        total_loss += loss.item()\n",
    "        \n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "    total_loss /= i+1\n",
    "    loss_history.append(total_loss)\n",
    "\n",
    "### TEST \n",
    "model.eval()\n",
    "loss = 0\n",
    "with torch.no_grad():\n",
    "    h, c = None, None\n",
    "    for i, window in enumerate(test_dataset):\n",
    "\n",
    "        y_pred, H, C = model(window, h, c)\n",
    "\n",
    "        h = H[0]\n",
    "        c = C[0]\n",
    "        \n",
    "        assert y_pred.shape == window.y.shape\n",
    "        loss += torch.mean((y_pred - window.y)**2)\n",
    "    loss /= i+1\n",
    "\n",
    "### RESULTS PLOT\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "x_ticks = np.arange(1, epochs+1)\n",
    "ax.plot(x_ticks, loss_history)\n",
    "\n",
    "# figure labels\n",
    "ax.set_title('Loss over time', fontweight='bold')\n",
    "ax.set_xlabel('Epochs', fontweight='bold')\n",
    "ax.set_ylabel('Mean Squared Error', fontweight='bold')\n",
    "plt.show()\n",
    "\n",
    "print(\"Train MSE: {:.4f}\".format(total_loss))\n",
    "print(\"Test MSE: {:.4f}\".format(loss))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "822b30e0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9dc1092468b6c95b716c12f4a89881ec47ccb10a5d5fac9cc7256670f9aa8e17"
  },
  "kernelspec": {
   "display_name": "Python 3.7.0 64-bit ('py37': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
